[ES|QL] Add support for `RRF` #221349

sddonne · 2025-05-23T10:11:46Z

Summary

Part of #215092

Considerations

`RRF` may change in the future

@ioanatia has stated that the command may change in the future as it's not fully implemented yet.

Licence check

Will be addressed at #216791

Screen.Recording.2025-05-23.at.12.42.15.mov

New command support checklist

AST support

Make sure that the new command is in the local Kibana grammar definition. The ANTLR lexer and parser files are updated every Monday from the source definition of the language at Elasticsearch (via a manually merged, automatically generated PR).
Create a factory for generating the new node. The new node should satisfy the ESQLCommand<Name> interface. If the syntax of your command cannot be decomposed only in parameters, you can hold extra information by extending the ESQLCommand interface. I.E., check the Rerank command.
While ANTLR is parsing the text query, we create our own AST by using onExit listeners at kbn-esql-ast/src/parser/esql_ast_builder_listener.ts. Implement the onExit<COMMAND_NAME> method based on the interface autogenerated by ANTLR and push the new node into the AST.
Create unit tests checking that the correct AST nodes are generated when parsing your command.
Add a dedicated visitor callback for the new command.
Verify that the Walker API can visit the new node.
Verify that the Synth API can construct the new node.

Validating that the command works well when prettifying the query

Validate that the prettifier works correctly.
Adjust the basic pretty printer and the wrapping pretty printer if needed.
Add unit tests validating that the queries are correctly formatted (even if no adjustment has been done).

Creating the command definition

Add the definition of the new command at kbn-esql-validation-autocomplete/src/definitions/commands.ts.

Adding the corresponding client-side validations

Add a custom validation if needed.
Add tests checking the behavior of the validation following this guide.

Adding the autocomplete suggestions

Add the suggestions to be shown when positioned at the new command.
- Create a new folder at kbn-esql-validation-autocomplete/src/autocomplete/commands for your command.
- Export a suggest function that should return an array of suggestions and set it up into the command definition.
- Optionally, we must filter or incorporate fields suggestions after a command is executed, this is supported by adding the fieldsSuggestionsAfter method to the command definition. Read this documentation to understand how it works.
If the new command must be suggested only in particular situations, modify the corresponding suggestions to make it possible.
Add tests following this guide.

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
Flaky Test Runner was used on any tests changed
The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

See some risk examples
...

elasticmachine · 2025-05-23T11:41:49Z

Pinging @elastic/kibana-esql (Team:ESQL)

sddonne · 2025-05-23T11:44:11Z

@elasticmachine merge upstream

stratoula

Looks great, added some comments.

Also RRF is on snapshot releases (which means we are not ready to suggest them). With your changes it is going to appear and we don't want it.

I am not sure if utilizing the hidden flag is a correct decision here. Ideally I want to have hidden true if it is still on snaphsots and find another way to hide it when it makes sense only after specific commands.

cc @drewdaemon

src/platform/packages/shared/kbn-esql-ast/src/parser/__tests__/rrf.test.ts

src/platform/packages/shared/kbn-esql-validation-autocomplete/src/__tests__/helpers.ts

...bn-esql-validation-autocomplete/src/autocomplete/commands/fork/commands_suggestions_after.ts

src/platform/packages/shared/kbn-esql-validation-autocomplete/src/autocomplete/autocomplete.ts

...form/packages/shared/kbn-esql-validation-autocomplete/src/autocomplete/commands/rrf/index.ts

src/platform/packages/shared/kbn-esql-validation-autocomplete/src/definitions/commands.ts

...atform/packages/shared/kbn-esql-validation-autocomplete/src/validation/commands/rrf/index.ts

into esql/rrf/ast-parsing

drewdaemon · 2025-05-23T17:59:53Z

I am not sure if utilizing the hidden flag is a correct decision here. Ideally I want to have hidden true if it is still on snaphsots and find another way to hide it when it makes sense only after specific commands.

This makes me wonder again if we should be treating RRF-after-FORK as a special case instead of trying to build a generic system for this.

I don't think it's a bad system @sddonne has set up here, but the autocomplete refactor was supposed to fix the anti-pattern of adding "generic" properties to the command definitions to support quirks only found in a single command. If we just handled RRF as a one-off in the top-level suggest method it might

be more clear what is actually going on
avoid growing the command definition interface
avoid using hidden in this new way

into esql/rrf/ast-parsing

stratoula

This looks great! Thanx Sebastian for taking care of the feedback. I left 2 comments but I am approving as I don't need to check them again.

...-esql-validation-autocomplete/src/validation/__tests__/test_suites/validation.command.rrf.ts

drewdaemon

This is great. I left a few comments. I'm approving with the assumption that those will be taken care of before merging.

src/platform/packages/shared/kbn-esql-validation-autocomplete/src/__tests__/helpers.ts

...kbn-esql-validation-autocomplete/src/autocomplete/__tests__/autocomplete.command.rff.test.ts

...-esql-validation-autocomplete/src/validation/__tests__/test_suites/validation.command.rrf.ts

Co-authored-by: Drew Tate <[email protected]>

elasticmachine · 2025-05-30T08:16:45Z

💚 Build Succeeded

Buildkite Build
Commit: 6c50dd2

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`esql`	227	229	+2
`lists`	454	456	+2
`securitySolution`	7450	7452	+2
`stackAlerts`	281	283	+2
`unifiedSearch`	388	390	+2
total			+10

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/esql-validation-autocomplete`	163	164	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`kbnUiSharedDeps-srcJs`	3.7MB	3.7MB	+1.5KB

Unknown metric groups

API count

id	before	after	diff
`@kbn/esql-validation-autocomplete`	184	185	+1

History

💔 Build #302590 failed 17ad55c
💔 Build #302358 failed c4a589e
💔 Build #302312 failed bb834fd
💛 Build #302080 was flaky 37b15c7
💔 Build #302023 failed 0779540

kibanamachine · 2025-05-30T08:21:37Z

Starting backport for target branches: 8.19

https://github.com/elastic/kibana/actions/runs/15342485445

kibanamachine · 2025-05-30T08:27:48Z

💔 All backports failed

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 221349

Questions ?

Please refer to the Backport tool documentation

sddonne · 2025-05-30T09:19:27Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

@ioanatia

## Summary Part of elastic#215092 ## Considerations ### `RRF` may change in the future @ioanatia has stated that the command may change in the future as it's not fully implemented yet. ### Licence check Will be addressed at elastic#216791 https://github.com/user-attachments/assets/759470a2-4afa-4d20-adbf-cb3001895e95 ## New command support checklist ### AST support - [x] Make sure that the new command is in the local Kibana grammar definition. The ANTLR lexer and parser files are updated every Monday from the source definition of the language at Elasticsearch (via a manually merged, automatically generated [PR](elastic#213006)). - [x] Create a factory for generating the new node. The new node should satisfy the `ESQLCommand<Name>` interface. If the syntax of your command cannot be decomposed only in parameters, you can hold extra information by extending the `ESQLCommand` interface. I.E., check the Rerank command. - [x] While ANTLR is parsing the text query, we create our own AST by using `onExit` listeners at `kbn-esql-ast/src/parser/esql_ast_builder_listener.ts`. Implement the `onExit<COMMAND_NAME>` method based on the interface autogenerated by ANTLR and push the new node into the AST. - [x] Create unit tests checking that the correct AST nodes are generated when parsing your command. - [x] Add a dedicated [visitor callback](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-esql-ast/src/visitor/README.md) for the new command. - [x] Verify that the `Walker` API can visit the new node. - [x] Verify that the `Synth` API can construct the new node. ### Validating that the command works well when prettifying the query - [x] Validate that the prettifier works correctly. - [ ] Adjust the basic pretty printer and the wrapping pretty printer if needed. - [x] Add unit tests validating that the queries are correctly formatted (even if no adjustment has been done). ### Creating the command definition - [x] Add the definition of the new command at `kbn-esql-validation-autocomplete/src/definitions/commands.ts`. ### Adding the corresponding client-side validations - [x] Add a custom validation if needed. - [x] Add tests checking the behavior of the validation following this [guide](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-esql-validation-autocomplete/README.md#the-new-way). ### Adding the autocomplete suggestions - [x] Add the suggestions to be shown when **positioned at** the new command. - [x] Create a new folder at `kbn-esql-validation-autocomplete/src/autocomplete/commands` for your command. - [x] Export a `suggest` function that should return an array of suggestions and set it up into the command definition. - [ ] Optionally, we must filter or incorporate fields suggestions after a command is executed, this is supported by adding the `fieldsSuggestionsAfter` method to the command definition. Read this documentation to understand how it works. - [x] If the new command must be suggested only in particular situations, modify the corresponding suggestions to make it possible. - [x] Add tests following this [guide](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-esql-validation-autocomplete/README.md#automated-testing). ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Stratoula Kalafateli <[email protected]> Co-authored-by: Drew Tate <[email protected]> (cherry picked from commit 0794030) # Conflicts: # src/platform/packages/shared/kbn-esql-validation-autocomplete/src/definitions/commands.ts

@ioanatia

## Summary Part of elastic#215092 ## Considerations ### `RRF` may change in the future @ioanatia has stated that the command may change in the future as it's not fully implemented yet. ### Licence check Will be addressed at elastic#216791 https://github.com/user-attachments/assets/759470a2-4afa-4d20-adbf-cb3001895e95 ## New command support checklist ### AST support - [x] Make sure that the new command is in the local Kibana grammar definition. The ANTLR lexer and parser files are updated every Monday from the source definition of the language at Elasticsearch (via a manually merged, automatically generated [PR](elastic#213006)). - [x] Create a factory for generating the new node. The new node should satisfy the `ESQLCommand<Name>` interface. If the syntax of your command cannot be decomposed only in parameters, you can hold extra information by extending the `ESQLCommand` interface. I.E., check the Rerank command. - [x] While ANTLR is parsing the text query, we create our own AST by using `onExit` listeners at `kbn-esql-ast/src/parser/esql_ast_builder_listener.ts`. Implement the `onExit<COMMAND_NAME>` method based on the interface autogenerated by ANTLR and push the new node into the AST. - [x] Create unit tests checking that the correct AST nodes are generated when parsing your command. - [x] Add a dedicated [visitor callback](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-esql-ast/src/visitor/README.md) for the new command. - [x] Verify that the `Walker` API can visit the new node. - [x] Verify that the `Synth` API can construct the new node. ### Validating that the command works well when prettifying the query - [x] Validate that the prettifier works correctly. - [ ] Adjust the basic pretty printer and the wrapping pretty printer if needed. - [x] Add unit tests validating that the queries are correctly formatted (even if no adjustment has been done). ### Creating the command definition - [x] Add the definition of the new command at `kbn-esql-validation-autocomplete/src/definitions/commands.ts`. ### Adding the corresponding client-side validations - [x] Add a custom validation if needed. - [x] Add tests checking the behavior of the validation following this [guide](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-esql-validation-autocomplete/README.md#the-new-way). ### Adding the autocomplete suggestions - [x] Add the suggestions to be shown when **positioned at** the new command. - [x] Create a new folder at `kbn-esql-validation-autocomplete/src/autocomplete/commands` for your command. - [x] Export a `suggest` function that should return an array of suggestions and set it up into the command definition. - [ ] Optionally, we must filter or incorporate fields suggestions after a command is executed, this is supported by adding the `fieldsSuggestionsAfter` method to the command definition. Read this documentation to understand how it works. - [x] If the new command must be suggested only in particular situations, modify the corresponding suggestions to make it possible. - [x] Add tests following this [guide](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-esql-validation-autocomplete/README.md#automated-testing). ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Stratoula Kalafateli <[email protected]> Co-authored-by: Drew Tate <[email protected]> (cherry picked from commit 0794030) # Conflicts: # src/platform/packages/shared/kbn-esql-validation-autocomplete/src/definitions/commands.ts

sddonne · 2025-05-30T09:43:05Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

@ioanatia

## Summary Part of elastic#215092 ## Considerations ### `RRF` may change in the future @ioanatia has stated that the command may change in the future as it's not fully implemented yet. ### Licence check Will be addressed at elastic#216791 https://github.com/user-attachments/assets/759470a2-4afa-4d20-adbf-cb3001895e95 ## New command support checklist ### AST support - [x] Make sure that the new command is in the local Kibana grammar definition. The ANTLR lexer and parser files are updated every Monday from the source definition of the language at Elasticsearch (via a manually merged, automatically generated [PR](elastic#213006)). - [x] Create a factory for generating the new node. The new node should satisfy the `ESQLCommand<Name>` interface. If the syntax of your command cannot be decomposed only in parameters, you can hold extra information by extending the `ESQLCommand` interface. I.E., check the Rerank command. - [x] While ANTLR is parsing the text query, we create our own AST by using `onExit` listeners at `kbn-esql-ast/src/parser/esql_ast_builder_listener.ts`. Implement the `onExit<COMMAND_NAME>` method based on the interface autogenerated by ANTLR and push the new node into the AST. - [x] Create unit tests checking that the correct AST nodes are generated when parsing your command. - [x] Add a dedicated [visitor callback](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-esql-ast/src/visitor/README.md) for the new command. - [x] Verify that the `Walker` API can visit the new node. - [x] Verify that the `Synth` API can construct the new node. ### Validating that the command works well when prettifying the query - [x] Validate that the prettifier works correctly. - [ ] Adjust the basic pretty printer and the wrapping pretty printer if needed. - [x] Add unit tests validating that the queries are correctly formatted (even if no adjustment has been done). ### Creating the command definition - [x] Add the definition of the new command at `kbn-esql-validation-autocomplete/src/definitions/commands.ts`. ### Adding the corresponding client-side validations - [x] Add a custom validation if needed. - [x] Add tests checking the behavior of the validation following this [guide](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-esql-validation-autocomplete/README.md#the-new-way). ### Adding the autocomplete suggestions - [x] Add the suggestions to be shown when **positioned at** the new command. - [x] Create a new folder at `kbn-esql-validation-autocomplete/src/autocomplete/commands` for your command. - [x] Export a `suggest` function that should return an array of suggestions and set it up into the command definition. - [ ] Optionally, we must filter or incorporate fields suggestions after a command is executed, this is supported by adding the `fieldsSuggestionsAfter` method to the command definition. Read this documentation to understand how it works. - [x] If the new command must be suggested only in particular situations, modify the corresponding suggestions to make it possible. - [x] Add tests following this [guide](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-esql-validation-autocomplete/README.md#automated-testing). ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Stratoula Kalafateli <[email protected]> Co-authored-by: Drew Tate <[email protected]> (cherry picked from commit 0794030)

sddonne · 2025-06-02T07:50:04Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

# Backport This will backport the following commits from `main` to `8.19`: - [[ES|QL] Add support for `RRF` (#221349)](#221349)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Stratoula Kalafateli <[email protected]> Co-authored-by: Drew Tate <[email protected]>

@ioanatia

## Summary Part of elastic#215092 ## Considerations ### `RRF` may change in the future @ioanatia has stated that the command may change in the future as it's not fully implemented yet. ### Licence check Will be addressed at elastic#216791 https://github.com/user-attachments/assets/759470a2-4afa-4d20-adbf-cb3001895e95 ## New command support checklist ### AST support - [x] Make sure that the new command is in the local Kibana grammar definition. The ANTLR lexer and parser files are updated every Monday from the source definition of the language at Elasticsearch (via a manually merged, automatically generated [PR](elastic#213006)). - [x] Create a factory for generating the new node. The new node should satisfy the `ESQLCommand<Name>` interface. If the syntax of your command cannot be decomposed only in parameters, you can hold extra information by extending the `ESQLCommand` interface. I.E., check the Rerank command. - [x] While ANTLR is parsing the text query, we create our own AST by using `onExit` listeners at `kbn-esql-ast/src/parser/esql_ast_builder_listener.ts`. Implement the `onExit<COMMAND_NAME>` method based on the interface autogenerated by ANTLR and push the new node into the AST. - [x] Create unit tests checking that the correct AST nodes are generated when parsing your command. - [x] Add a dedicated [visitor callback](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-esql-ast/src/visitor/README.md) for the new command. - [x] Verify that the `Walker` API can visit the new node. - [x] Verify that the `Synth` API can construct the new node. ### Validating that the command works well when prettifying the query - [x] Validate that the prettifier works correctly. - [ ] Adjust the basic pretty printer and the wrapping pretty printer if needed. - [x] Add unit tests validating that the queries are correctly formatted (even if no adjustment has been done). ### Creating the command definition - [x] Add the definition of the new command at `kbn-esql-validation-autocomplete/src/definitions/commands.ts`. ### Adding the corresponding client-side validations - [x] Add a custom validation if needed. - [x] Add tests checking the behavior of the validation following this [guide](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-esql-validation-autocomplete/README.md#the-new-way). ### Adding the autocomplete suggestions - [x] Add the suggestions to be shown when **positioned at** the new command. - [x] Create a new folder at `kbn-esql-validation-autocomplete/src/autocomplete/commands` for your command. - [x] Export a `suggest` function that should return an array of suggestions and set it up into the command definition. - [ ] Optionally, we must filter or incorporate fields suggestions after a command is executed, this is supported by adding the `fieldsSuggestionsAfter` method to the command definition. Read this documentation to understand how it works. - [x] If the new command must be suggested only in particular situations, modify the corresponding suggestions to make it possible. - [x] Add tests following this [guide](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-esql-validation-autocomplete/README.md#automated-testing). ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Stratoula Kalafateli <[email protected]> Co-authored-by: Drew Tate <[email protected]>

@ioanatia

## Summary Part of elastic#215092 ## Considerations ### `RRF` may change in the future @ioanatia has stated that the command may change in the future as it's not fully implemented yet. ### Licence check Will be addressed at elastic#216791 https://github.com/user-attachments/assets/759470a2-4afa-4d20-adbf-cb3001895e95 ## New command support checklist ### AST support - [x] Make sure that the new command is in the local Kibana grammar definition. The ANTLR lexer and parser files are updated every Monday from the source definition of the language at Elasticsearch (via a manually merged, automatically generated [PR](elastic#213006)). - [x] Create a factory for generating the new node. The new node should satisfy the `ESQLCommand<Name>` interface. If the syntax of your command cannot be decomposed only in parameters, you can hold extra information by extending the `ESQLCommand` interface. I.E., check the Rerank command. - [x] While ANTLR is parsing the text query, we create our own AST by using `onExit` listeners at `kbn-esql-ast/src/parser/esql_ast_builder_listener.ts`. Implement the `onExit<COMMAND_NAME>` method based on the interface autogenerated by ANTLR and push the new node into the AST. - [x] Create unit tests checking that the correct AST nodes are generated when parsing your command. - [x] Add a dedicated [visitor callback](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-esql-ast/src/visitor/README.md) for the new command. - [x] Verify that the `Walker` API can visit the new node. - [x] Verify that the `Synth` API can construct the new node. ### Validating that the command works well when prettifying the query - [x] Validate that the prettifier works correctly. - [ ] Adjust the basic pretty printer and the wrapping pretty printer if needed. - [x] Add unit tests validating that the queries are correctly formatted (even if no adjustment has been done). ### Creating the command definition - [x] Add the definition of the new command at `kbn-esql-validation-autocomplete/src/definitions/commands.ts`. ### Adding the corresponding client-side validations - [x] Add a custom validation if needed. - [x] Add tests checking the behavior of the validation following this [guide](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-esql-validation-autocomplete/README.md#the-new-way). ### Adding the autocomplete suggestions - [x] Add the suggestions to be shown when **positioned at** the new command. - [x] Create a new folder at `kbn-esql-validation-autocomplete/src/autocomplete/commands` for your command. - [x] Export a `suggest` function that should return an array of suggestions and set it up into the command definition. - [ ] Optionally, we must filter or incorporate fields suggestions after a command is executed, this is supported by adding the `fieldsSuggestionsAfter` method to the command definition. Read this documentation to understand how it works. - [x] If the new command must be suggested only in particular situations, modify the corresponding suggestions to make it possible. - [x] Add tests following this [guide](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-esql-validation-autocomplete/README.md#automated-testing). ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Stratoula Kalafateli <[email protected]> Co-authored-by: Drew Tate <[email protected]>

sddonne added 11 commits May 20, 2025 15:07

feat: add rrf node

e1d67fb

feat: add rff ast parser tests

83b0e3e

feat: add rrf to visitor api

e3b0da1

feat: add tests for rrf pretty printer

084f04b

feat: add new rff command and suggestions

d19f0bd

feat: only display RRF command after a FORK

06a937e

feat: only display RRF command after FORK

832a5fa

feat: add tests for rrf autocomplete

39d8b2d

feat: add rrf validations and autocomplete tests

36dd3c4

feat: add validations for rrf command

fac9e5c

fix: typo

89b6516

sddonne added Feature:ES|QL ES|QL related features in Kibana Team:ESQL ES|QL related features in Kibana backport:skip This commit does not require backporting v9.1.0 release_note:skip Skip the PR/issue when compiling release notes labels May 23, 2025

sddonne marked this pull request as ready for review May 23, 2025 11:41

sddonne requested a review from a team as a code owner May 23, 2025 11:41

sddonne requested review from drewdaemon, stratoula and vadimkibana May 23, 2025 11:42

Merge branch 'main' into esql/rrf/ast-parsing

0779540

stratoula reviewed May 23, 2025

View reviewed changes

sddonne added 4 commits May 23, 2025 15:59

add failure test to rrf command parse

80655ce

Merge branch 'esql/rrf/ast-parsing' of https://github.com/sddonne/kibana

36fa32d

into esql/rrf/ast-parsing

fix: parameter not needed

2130458

unify metada error codes

37b15c7

sddonne and others added 2 commits May 26, 2025 13:47

Merge branch 'esql/rrf/ast-parsing' of https://github.com/sddonne/kibana

c4a589e

into esql/rrf/ast-parsing

Merge branch 'main' into esql/rrf/ast-parsing

c49ed97

stratoula approved these changes May 27, 2025

View reviewed changes

...-esql-validation-autocomplete/src/validation/__tests__/test_suites/validation.command.rrf.ts Outdated Show resolved Hide resolved

sddonne added 2 commits May 27, 2025 11:34

add rrf test case, when invoked with arguments

1d46db4

fix rrf tests formatting

17ad55c

sddonne removed the request for review from vadimkibana May 27, 2025 11:06

drewdaemon approved these changes May 27, 2025

View reviewed changes

sddonne and others added 4 commits May 28, 2025 10:07

ignore case sensitivity in test regex

bdf3b6e

Co-authored-by: Drew Tate <[email protected]>

fix wrong test suite name

509ae43

fix: rrf autosuggest test

0e16d3d

Merge branch 'main' into esql/rrf/ast-parsing

6c50dd2

sddonne merged commit 0794030 into elastic:main May 30, 2025
10 checks passed

sddonne mentioned this pull request May 30, 2025

[8.19] [ES|QL] Add support for RRF (#221349) #222020

Closed

sddonne mentioned this pull request May 30, 2025

[8.19] [ES|QL] Add support for RRF (#221349) #222023

Closed

This was referenced May 30, 2025

[TSVB/SCSS] Migrate all remaining .scss styles to Emotion CSS-in-JS #221484

Merged

[ES|QL] support SAMPLE command #221446

Merged

sddonne mentioned this pull request Jun 2, 2025

[8.19] [ES|QL] Add support for RRF (#221349) #222147

Merged

stratoula mentioned this pull request Jun 2, 2025

[ES|QL] Support the new RRF command #215092

Closed

[ES|QL] Add support for RRF #221349

[ES|QL] Add support for RRF #221349

Uh oh!

Conversation

sddonne commented May 23, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Considerations

RRF may change in the future

Licence check

New command support checklist

AST support

Validating that the command works well when prettifying the query

Creating the command definition

Adding the corresponding client-side validations

Adding the autocomplete suggestions

Checklist

Identify risks

Uh oh!

elasticmachine commented May 23, 2025

Uh oh!

sddonne commented May 23, 2025

Uh oh!

stratoula left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

drewdaemon commented May 23, 2025

Uh oh!

stratoula left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

drewdaemon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented May 30, 2025

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Page load bundle

API count

History

Uh oh!

Uh oh!

kibanamachine commented May 30, 2025

Uh oh!

kibanamachine commented May 30, 2025

💔 All backports failed

Manual backport

Questions ?

Uh oh!

sddonne commented May 30, 2025

💚 All backports created successfully

Questions ?

Uh oh!

sddonne commented May 30, 2025

💚 All backports created successfully

Questions ?

Uh oh!

sddonne commented Jun 2, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Uh oh!

[ES|QL] Add support for `RRF` #221349

[ES|QL] Add support for `RRF` #221349

sddonne commented May 23, 2025 •

edited by kibanamachine

Loading

`RRF` may change in the future